Persist real face embeddings during actor indexing (unblocks #74) - #151
Persist real face embeddings during actor indexing (unblocks #74)#151ik020 wants to merge 1 commit into
Conversation
prep) Actor indexing computed a normalized face encoding per detection but never stored it: _actor_records() wrote a placeholder embedding=[0.0] into every StorageRecord, so the actor Chroma collection's vector index has never contained anything searchable. Matching only ever happened in-memory, within a single indexing run, via ActorIndexState.known_encodings, and was discarded once that run finished. This blocks issue grayhatdevelopers#74 (find people using a reference image), which requires comparing a reference image's embedding against previously indexed faces across the whole repository. Fix: - Carry the computed encoding through into each detection dict. - _actor_records() now writes the real normalized encoding as the StorageRecord embedding instead of the placeholder. - Cluster-summary records (_actor_cluster_records) are left unchanged: a summary rolls up multiple detections and has no single face image to encode. Because this changes what's actually stored per detection, any existing generation with the actor modality enabled has placeholder vectors and is no longer valid. Bump INDEX_SCHEMA_VERSION 7 -> 8 so CompletedGenerationManifest's Literal[INDEX_SCHEMA_VERSION] check rejects old generations with a clear IndexSchemaError instead of silently treating placeholder-vector indexes as complete and searchable. Affected generations need to be re-indexed. Adds a new end-to-end test driving process_actor_samples through mocked detector/recognizer calls, asserting the stored embedding is the real normalized encoding rather than [0.0]. Updates the existing _actor_records test, which built detection dicts without the now-required encoding key.
|
Hey @ik020, |
Thanks! I’ve already forked the repo and have been working on my branch. I’ve spent the past week working on it and have fixed 4 issues with PRs submitted. Just to clarify, I’m not an autonomous agent 😄,I’m a final-year CS undergraduate at FAST NUCES Islamabad. Our university shared this opportunity with us via email, which is how I found the repo and started contributing. I’ll star the repo as well. Thanks! |
Prep Work for #74 — Find People Using a Reference Image
The Bug
Actor indexing detects a face, aligns it, and computes a normalized SFace encoding for each detection — but that encoding was never actually persisted.
_actor_records()was writing a placeholderembedding=[0.0]into everyStorageRecordsent to Chroma. As a result, theactorcollection's vector index has never contained real, searchable face embeddings.During indexing, matching only used
ActorIndexState.known_encodings, which is an in-memory list scoped to a single indexing run and discarded once the run finishes.This silently blocks #74: finding a person using a reference image requires comparing a new face embedding against all previously detected faces across the repository, but there was no durable embedding data to compare against.
The Fix
encodingthrough into each detection dictionary inprocess_actor_samples._actor_records()to persist the real, normalized encoding asStorageRecord.embeddinginstead of[0.0]._actor_cluster_records()unchanged. A cluster summary spans multiple detections and does not represent a single face image, so it has no single embedding to persist.Schema Version Bump
Because this changes what is actually persisted for each actor detection, existing generations with the
actormodality enabled contain placeholder vectors and are no longer trustworthy for face search.The repository already has a mechanism for handling this:
CompletedGenerationManifest.index_schema_versionis typed asLiteral[INDEX_SCHEMA_VERSION].local_snapshots.py::validate_generation()validates the manifest on disk usingmodel_validate_json().ValidationErroris explicitly caught and re-raised asIndexSchemaError.This PR bumps
INDEX_SCHEMA_VERSIONfrom 7 → 8.As a result, old generations will fail validation cleanly with
IndexSchemaErrorinstead of silently being treated as complete, searchable indexes containing meaningless vectors.User-facing impact
Any existing generation with the
actormodality enabled will need to be re-indexed after this change.This is intentional. The alternative would be allowing #74 to search against invalid placeholder embeddings with no indication that the underlying data is unusable.
Testing
New test
Added:
test_actor_indexing_persists_real_face_embeddings_not_placeholdersThis drives
process_actor_samplesthrough mocked YuNet/SFace calls end-to-end and verifies that theStorageRecordsent tostorage.upsertcontains the real normalized encoding rather than the placeholder[0.0].Updated test
Updated:
test_actor_records_preserve_stable_detection_metadataThe test previously created fake detection dictionaries without an
encodingfield. It now provides an explicit encoding and verifies that the resultingStorageRecord.embeddingmatches the real value.Test results
Full test suite:
590 passed, 2 skipped, 0 failed
Also re-ran the tests covering
INDEX_SCHEMA_VERSIONand actor indexing:test_generation_manifest.pytest_local_snapshots.pytest_indexing.pytest_actor_results.pytest_cli.pytest_frontend_app.pyAll tests pass, including the existing test that constructs a manifest with
INDEX_SCHEMA_VERSION + 1to verify that the rejection path still works.Not Included
This PR does not implement reference-image search itself.
Reference-image search remains the actual feature tracked by #74 and will be implemented as a separate follow-up PR once this change lands, since it depends on real actor embeddings being persisted first.